System for fully integrated capture, and analysis of business information resulting in predictive decision making and simulation

ABSTRACT

A system for fully integrated collection of business impacting data, analysis of that data and generation of both analysis driven business decisions and analysis driven simulations of alternate candidate business action comprising a business data retrieval engine stored in a memory of and operating on a processor of a computing device, a business data analysis engine stored in a memory of and operating on a processor of a computing device and a business decision and business action path simulation engine stored in a memory of and operating on a processor of one of more computing devices has been developed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/925,974, titled “RAPID PREDICTIVE ANALYSIS OF VERY LARGEDATA SETS USING THE DISTRIBUTED COMPUTATIONAL GRAPH” and filed on Oct.28, 2015, and is also a continuation-in-part of U.S. patent applicationSer. No. 14/986,536, titled “DISTRIBUTED SYSTEM FOR LARGE VOLUME DEEPWEB DATA EXTRACTION”, and filed on Dec. 31, 2015, and is also acontinuation-in-part of U.S. patent application Ser. No. 15/091,563,titled “SYSTEM FOR CAPTURE, ANALYSIS AND STORAGE OF TIME SERIES DATAFROM SENSORS WITH HETEROGENEOUS REPORT INTERVAL PROFILES”, and filed onApr. 5, 2016, the entire specification of each of which is incorporatedherein by reference in its entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention is in the field of use of computer systems inbusiness information management, operations and predictive planning.Specifically, the development of a system that integrates the functionsof business information and operating data, complex data analysis anduse of that data, preprogrammed commands and parameters and machinelearning to create a business operating system capable of predictivedecision making and action path outcome simulation.

Discussion of the State of the Art

Over the past decade the amount of financial, operational,infrastructure, risk management and philosophical information availableto decision makers of a business from such sources as ubiquitous sensorsfound on a business's equipment or available from third party sources,detailed cause and effect data, and business process monitoring softwarehas expanded to the point where the data has overwhelmed the abilitiesof virtually anyone to follow all of it much less interpret and makemeaningful use of that available data in a given business environment.In other words the torrent of business related information now availableto a decision maker of group of decision makers has far out grown theability of those in most need of its use to either fully follow it orreliably use it. Failure to recognize important trends or become awareof information in a timely fashion has led to highly visible, customerfacing, outages at NETFLIX™, FACEBOOK™, and UPS™ over the past fewyears, just to list a few.

There have several been developments in business software that havearisen with the purpose of streamlining or automating either businessdata analysis or business decision process. PALANTIR™ offers software toisolate patterns in large volumes of data, DATABRICKS™ offers customanalytics services ANAPLAN™ offers financial impact calculation servicesand there are other software sources that mitigate some aspect ofbusiness data relevancy identification, analysis of that data andbusiness decision automation, but none of these solutions handle morethan a single aspect of the whole task.

What is needed is a fully integrated system that retrieves businessrelevant information from many diverse sources, identifies and analyzesthat high volume data, transforming it to a business useful format andthen uses that data to create intelligent predictive business decisionsand business pathway simulations. Forming a “business operating system.”

SUMMARY OF THE INVENTION

Accordingly, the inventor has developed a distributed system for thefully integrated retrieval, and deep analysis of business operationalinformation from a plurality of sources. The system further uses resultsof business information analytics to optimize the making of businessdecisions and allow for alternate action pathways to be simulated usingthe latest data and machine mediated prediction algorithms.

According to a preferred embodiment of the invention, a system forcomprising: a business data retrieval engine stored in a memory of andoperating on a processor of a computing device, a business data analysisengine stored in a memory of and operating on a processor of a computingdevice and a business decision and business action path simulationengine stored in a memory of and operating on a processor of one of morecomputing devices. Wherein, the business information retrieval engine:retrieves a plurality of business related data from a plurality ofsources, accepts a plurality of analysis parameters and control commandsdirectly from human interface devices or from one or more command andcontrol storage devices, and stores accumulated retrieved informationfor processing by data analysis engine or predetermined data timeout.The business information analysis engine: retrieves a plurality of datatypes from the business information retrieval engine, performs aplurality of analytical functions and transformations on retrieved databased upon the specific goals and needs set forth in a current campaignby business process analysis authors. Wherein the business decision andbusiness action path simulation engine: employs results of data analysesand transformations performed by the business information analysisengine, together with available supplemental data from a plurality ofsources as well as any current campaign specific machine learning,commands and parameters from business process analysis authors toformulate current business operations and risk status reports andemploys results of data analyses and transformations performed by thebusiness information analysis engine, together with availablesupplemental data from a plurality of sources, any current campaignspecific commands and parameters from business process analysis authors,as well as input gleaned from machine learning algorithms to deliverbusiness action pathway simulations and business decision support to afirst end user.

According to another embodiment of the invention, the system's businessinformation retrieval engine is stored in the memory of and operating ona processor of a computing device, employs a portal for human interfacedevice input at least a portion of which are business related data andat least another portion of which are commands and parameters related tothe conduct of a current business analysis campaign. The businessinformation retrieval engine employs a high throughput deep web scraperstored in the memory of an operating on a processor of a computingdevice, which receives at least some spider configuration parametersfrom the highly customizable cloud based interface, coordinates one ormore world wide web searches (scrapes) using both general search controlparameters and individual web search agent (spider) specificconfiguration data, receives scrape progress feedback information whichmay lead to issuance of further web search control parameters, controlsand monitors the spiders on distributed scrape servers, receives the rawscrape campaign data from scrape servers, aggregates at least portionsof scrape campaign data from each web site or web page traversed as perthe parameters of the scrape campaign. The archetype spiders areprovided by a program library and individual spiders are created usingconfiguration files. Scrape campaign requests are persistently storedand can be reused or used as the basis for similar scrape campaigns. Thebusiness information retrieval engine employs a multidimensional timeseries data store stored in a memory of and operating on a processor ofa computing device to receive a plurality of data from a plurality ofsensors of heterogeneous types, some of which may have heterogeneousreporting and data payload transmission profiles, aggregates the sensordata over a predetermined amount of time, a predetermined quantity ofdata or a predetermined number of events, retrieves a specific quantityof aggregated sensor data per each access connection predetermined toallow reliable receipt and inclusion of the data, transparentlyretrieves quantities of aggregated sensor data too large to be reliablytransferred by one access connection using a further plurality accessconnections to allow capture of all aggregated sensor data underconditions of heavy sensor data influx and stores aggregated sensor datain a simple key-value pair with very little or no data transformationfrom how the aggregated sensor data is received. Last, the business dataanalysis engine employs a directed computational graph stored in thememory of an operating system on a processor of a computing devicewhich, retrieves streams of input from one or more of a plurality ofdata sources, filters data to remove data records from the stream for aplurality of reasons drawn from, but not limited to a set comprisingabsence of all information, damage to data in the record, and presenceof in-congruent information or missing information which invalidates thedata record, splits filtered data stream into two or more identicalparts, formats data within one data stream based upon a set ofpredetermined parameters so as to prepare for meaningful storage in adata store, sends identical data stream further analysis and eitherlinear transformation or branching transformation using resources of thesystem.

According to another embodiment of the invention, a method for fullyintegrated capture, and transformative analysis of business impactfulinformation resulting in predictive decision making and simulation themethod comprising the steps of: (a) retrieving business related data andanalysis campaign command and control information using a businessinformation retrieval engine stored in the memory of an operating on aprocessor of a computing device; (b) analyzing and transformingretrieved business related data using a business information analysisengine stored in the memory of an operating on a processor of acomputing device in conjunction with previously designed analysiscampaign command and control information; and (c) presenting businessdecision critical information as well as business action pathwaysimulation information using a business decision and business actionpath simulation engine based upon the results of analysis of previouslyretrieved business related data and previously entered analysis campaigncommand and control information.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. One skilled inthe art will recognize that the particular embodiments illustrated inthe drawings are merely exemplary, and are not intended to limit thescope of the present invention.

FIG. 1 is a diagram of an exemplary architecture of a business operatingsystem according to an embodiment of the invention.

FIG. 2 is a process flow diagram showing an exemplary set of steps usedin the function of the very high bandwidth cloud interface.

FIG. 3 is a diagram of an exemplary architecture for a lineartransformation pipeline system which introduces the concept of thetransformation pipeline as a directed graph of transformation nodes andmessages according to an embodiment of the invention.

FIG. 4 is a process flow diagram of a method for an embodiment ofmodeling the transformation pipeline module of the invention as adirected graph using graph theory.

FIG. 5 is a process flow diagram of a method for one embodiment of alinear transformation pipeline.

FIG. 6 is a process flow diagram of a method for one embodiment of atransformation pipeline where one transformation node in atransformation pipeline receives data streams from two sourcetransformation nodes.

FIG. 7 is a process flow diagram of a method for one embodiment of atransformation pipeline where one transformation node in atransformation pipeline sends output data stream to two destinationtransformation nodes in potentially two separate transformationpipelines.

FIG. 8 is a diagram showing exemplary world wide web target sitescontaining the type of loosely structured, large volume, data that makethem candidates for search and retrieval by the invention according toan embodiment of the invention.

FIG. 9 is a process flow diagram of a method for a high volume webcrawling module.

FIG. 10 is a listing of a very simple example Scrapy web spiderconfiguration file.

FIG. 11 is a method flow diagram showing an exemplary set of step usedin the capture and storage of time series data from sensors withheterogeneous reporting profiles according to an embodiment of theinvention.

FIG. 12 is a process flow diagram of a method for the use ofmetaswimlanes to transparently accommodate levels of data streamingwhich would overload a single swimlane according to an embodiment of theinvention.

FIG. 13 is a simplified example of the use a Kalman filter to extractand smooth estimated system state from noisy sensor data according to anembodiment of the invention.

FIG. 14 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

FIG. 15 is a block diagram illustrating an exemplary logicalarchitecture for a client device, according to various embodiments ofthe invention.

FIG. 16 is a block diagram illustrating an exemplary architecturalarrangement of clients, servers, and external services, according tovarious embodiments of the invention.

FIG. 17 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention

FIG. 18 is a method process flow diagram showing the operation of anautomated planning service module according to an embodiment of theinvention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, a system and methodfor fully integrated capture and analysis of business informationresulting in predictive decision making and simulation.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beunderstood that these are presented for illustrative purposes only. Thedescribed embodiments are not intended to be limiting in any sense. Oneor more of the inventions may be widely applicable to numerousembodiments, as is readily apparent from the disclosure. In general,embodiments are described in sufficient detail to enable those skilledin the art to practice one or more of the inventions, and it is to beunderstood that other embodiments may be utilized and that structural,logical, software, electrical and other changes may be made withoutdeparting from the scope of the particular inventions. Accordingly,those skilled in the art will recognize that one or more of theinventions may be practiced with various modifications and alterations.Particular features of one or more of the inventions may be describedwith reference to one or more particular embodiments or figures thatform a part of the present disclosure, and in which are shown, by way ofillustration, specific embodiments of one or more of the inventions. Itshould be understood, however, that such features are not limited tousage in the one or more particular embodiments or figures withreference to which they are described. The present disclosure is neithera literal description of all embodiments of one or more of theinventions nor a listing of features of one or more of the inventionsthat must be present in all embodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring sequentially (e.g., because one step is described after theother step). Moreover, the illustration of a process by its depiction ina drawing does not imply that the illustrated process is exclusive ofother variations and modifications thereto, does not imply that theillustrated process or any of its steps are necessary to one or more ofthe invention(s), and does not imply that the illustrated process ispreferred. Also, steps are generally described once per embodiment, butthis does not mean they must occur once, or that they may only occuronce each time a process, method, or algorithm is carried out orexecuted. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described, it will be readily apparent that a single deviceor article may be used in place of the more than one device or article.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should be notedthat particular embodiments include multiple iterations of a techniqueor multiple manifestations of a mechanism unless noted otherwise.Process descriptions or blocks in figures should be understood asrepresenting modules, segments, or portions of code which include one ormore executable instructions for implementing specific logical functionsor steps in the process. Alternate implementations are included withinthe scope of embodiments of the present invention in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

DEFINITIONS

As used herein, a “swimlane” is a communication channel between a timeseries sensor data reception and apportioning device and a data storemeant to hold the apportioned data time series sensor data. A swimlaneis able to move a specific, finite amount of data between the twodevices. For example a single swimlane might reliably carry and haveincorporated into the data store, the data equivalent of 5 seconds worthof data from 10 sensors in 5 seconds, this being its capacity. Attemptsto place 5 seconds worth of data received from 6 sensors using oneswimlane would result in data loss.

As used herein, a “metaswimlane” is an as-needed logical combination oftransfer capacity of two or more real swimlanes that is transparent tothe requesting process. Sensor studies where the amount of data receivedper unit time is expected to be highly heterogeneous over time may beinitiated to use metaswimlanes. Using the example used above that asingle real swimlane can transfer and incorporate the 5 seconds worth ofdata of 10 sensors without data loss, the sudden receipt of incomingsensor data from 13 sensors during a 5 second interval would cause thesystem to create a two swimlane metaswimlane to accommodate the standard10 sensors of data in one real swimlane and the 3 sensor data overage inthe second, transparently added real swimlane, however no changes to thedata receipt logic would be needed as the data reception andapportionment device would add the additional real swimlanetransparently.

Conceptual Architecture

FIG. 1 is a diagram of an exemplary architecture of a business operatingsystem 100 according to an embodiment of the invention. Client access tothe system 105 both for system control and for interaction with systemoutput such as automated predictive decision making and planning andalternate pathway simulations, occurs through the system's highlydistributed, very high bandwidth cloud interface 110 which isapplication driven through the use of the Scala/Lift developmentenvironment and web interaction operation mediated by AWS ELASTICBEANSTALK™, both used for standards compliance and ease of development.Much of the business data analyzed by the system both from sourceswithin the confines of the client business, and from cloud basedsources, also enter the system through the cloud interface 110, databeing passed to the analysis and transformation components of thesystem, the directed computational graph module 155, high volume webcrawling module 115 and multidimensional time series database 120. Thedirected computational graph retrieves one or more streams of data froma plurality of sources, which includes, but is in no way not limited to,a number of physical sensors, web based questionnaires and surveys,monitoring of electronic infrastructure, crowd sourcing campaigns, andhuman input device information. Within the directed computational graph,data may be split into two identical streams, wherein one sub-stream maybe sent for batch processing and storage while the other sub-stream maybe reformatted for transformation pipeline analysis. The data is thentransferred to general transformer service 160 for linear datatransformation as part of analysis or decomposable transformer service150 for branching or iterative transformations that are part ofanalysis. The directed computational graph 155 represents all data asdirected graphs where the transformations are nodes and the resultmessages between transformations edges of the graph. These graphs whichcontain considerable intermediate transformation data are stored andfurther analyzed within graph stack module 145. High volume web crawlingmodule 115 uses multiple server hosted preprogrammed web spiders to findand retrieve data of interest from web based sources that are not welltagged by conventional web crawling technology. Multiple dimension timeseries database module 120 receives data from a large plurality ofsensors that may be of several different types. The module is designedto accommodate irregular and high volume surges by dynamically allottingnetwork bandwidth and server processing channels to process the incomingdata. Data retrieved by the multidimensional time series database 120and the high volume web crawling module 115 may be further analyzed andtransformed into task optimized results by the directed computationalgraph 155 and associated general transformer service 150 anddecomposable transformer service 160 modules.

Results of the transformative analysis process may then be combined withfurther client directives, additional business rules and practicesrelevant to the analysis and situational information external to thealready available data in the automated planning service module 130which also runs powerful predictive statistics functions and machinelearning algorithms to allow future trends and outcomes to be rapidlyforecast based upon the current system derived results and choosing eacha plurality of possible business decisions. Using all available data,the automated planning service module 130 may propose business decisionsmost likely to result is the most favorable business outcome with ausably high level of certainty. Closely related to the automatedplanning service module in the use of system derived results inconjunction with possible externally supplied additional information inthe assistance of end user business decision making, the businessoutcome simulation module 125 coupled with the end user facingobservation and state estimation service 140 allows business decisionmakers to investigate the probable outcomes of choosing one pendingcourse of action over another based upon analysis of the currentavailable data. For example, the pipelines operations department hasreported a very small reduction in crude oil pressure in a section ofpipeline in a highly remote section of territory. Many believe the issueis entirely due to a fouled, possibly failing flow sensor, othersbelieve that it is a proximal upstream pump that may have foreignmaterial stuck in it. Correction of both of these possibilities is toincrease the output of the effected pump to hopefully clean out it orthe fouled sensor. A failing sensor will have to be replaced at the nextmaintenance cycle. A few, however, feel that the pressure drop is due toa break in the pipeline, probably small at this point, but even so,crude oil is leaking and the remedy for the fouled sensor or pump optioncould make the leak much worse and waste much time afterwards. Thecompany does have a contractor about 8 hours away, or could rentsatellite time to look but both of those are expensive for a probablesensor issue, significantly less than cleaning up an oil spill thoughand then with significant negative public exposure. These sensor issueshave happened before and the business operating system 100 has data fromthem, which no one really studied due to the great volume of columnarfigures, so the alternative courses 125, 140 of action are run. Thesystem, based on all available data predicts that the fouled sensor orpump are unlikely the root cause this time due to other available dataand the contractor is dispatched. She finds a small breach in thepipeline. There will be a small cleanup and the pipeline needs to beshutdown for repair but multiple tens of millions of dollars have beensaved. This is just one example of a great many of the possible use ofthe business operating system, those knowledgeable in the art willeasily formulate more.

FIG. 2 is a process flow diagram showing an exemplary set of steps usedin the function of the very high bandwidth cloud interface 200, alsodepicted in FIG. 1, 110. Data flowing into and out of the very highbandwidth cloud interface 200 may come from human interactions throughdesktop or mobile computing devices 202, reading data sent from remotesensor arrays 203 and data retrieved from web pages 204 both of which203, 204 may reach a very high instantaneous volume for moderate timeintervals which must be accommodated by the interface to assure reliabledata capture. It should be noted that while the cloud 201 may usuallymean the internet, often the World Wide Web in the current context, italso extends here to data transmitted from the confines of the clientbusiness to the business operating system which may use a separatenetwork topology. Within the very high bandwidth cloud interface, webapps, constructed and supported using mostly open source resources,present graphical interfaces for end users to both submit newinformation 207 and to visualize the results of analyses and predictivedecisions as well as simulations created by the business operatingsystem 208. Programming is also used to accept and properly routecommand line directives and parameters from analysts and programmers tothe system as analyses are carried out 208. Sensor data and raw webpagedata being retrieved by the multiple dimension time series databasemodule, depicted in 120, and high volume web crawling module depicted in115, also may pass through the high volume interface 205. While thisembodiment represents the cloud interface as a monolithic portion of thebusiness operating system architecture, the invention has no suchrequirement and thus in other embodiments, data, programming command andcampaign parameters may enter the system from multiple portal to thecloud.

FIG. 3 is a block diagram of a preferred architecture for atransformation pipeline within a system for predictive analysis of verylarge data sets using distributed computational graph 300. According tothe embodiment, streaming input 315 serves as input to the firsttransformation node 320 of the transformation pipeline. Transformationnode's function is performed on input data stream and transformed outputmessage 325 is sent to transformation node 2 330. The progression oftransformation nodes 320, 330, 340, 350, 360 and associated outputmessages from each node 325, 335, 345, 355 is linear in configurationthis is the simplest arrangement and, as previously noted, representsthe current state of the art. While transformation nodes are describedaccording to various embodiments as uniform shape, such uniformity isused for presentation simplicity and clarity and does not reflectnecessary operational similarity between transformations within thepipeline. It should be appreciated that one knowledgeable in the fieldwill realize that certain transformations in a pipeline may be entirelyself-contained; certain transformations may involve human interactionthrough a program running on a desktop or mobile device 330, such asselection via dial or dials, positioning of switch or switches, orparameters set on control display, all of which may change duringanalysis; other transformations may require external aggregation orcorrelation services or may rely on remote procedure calls tosynchronous or asynchronous analysis engines as might occur insimulations among a plurality of other possibilities. Further accordingto the embodiment, individual transformation nodes in one pipeline mayrepresent function of another transformation pipeline. It should beappreciated that the node length of transformation pipelines depicted inno way confines the transformation pipelines employed by the inventionto an arbitrary maximum length 340, 350, 360 as, being distributed, thenumber of transformations would be limited by the resources madeavailable to each implementation of the invention. It should be furtherappreciated that there need be no limits on transform pipeline length.Output of the last transformation node and by extension, the transformpipeline 360 may be sent back to messaging software module 135 forpredetermined action.

FIG. 4 is a process flow diagram of a method 400 for an embodiment ofmodeling the transformation pipeline module 160 of the invention as adirected graph using graph theory 155. According to the embodiment, theindividual transformations 402, 404, 406 of the transformation pipelinet₁ . . . t_(n) such that each t_(i) T are represented as graph nodes.Transformations belonging to T are discrete transformations overindividual datasets d_(i), consistent with classical functions. As such,each individual transformation t_(j), receives a set of inputs andproduces a single output. The input of an individual transformationt_(i), is defined with the function in: t_(i) d₁ . . . d_(k) such thatin(t_(i))={d₁ . . . d_(k)) and describes a transformation with k inputs.Similarly, the output of an individual transformation is defined as thefunction out: t_(i) [ld₁] to describe transformations that produce asingle output (usable by other transformations). A dependency functioncan now be defined such that dep(t_(a),t_(b)) out(t_(a))in(t_(b)) Themessages carrying the data stream through the transformation pipeline401, 403, 405 make up the graph edges. Using the above definitions,then, a transformation pipeline within the invention can be defined asG=(V,E) where message(t₁,t₂ . . . t_(n))V and all transformations t₁ . .. t_(n) and all dependencies dep(t_(i),t_(j))E 407.

FIG. 5 is a process flow diagram of a method 500 for one embodiment of alinear transformation pipeline 501. This is the simplest ofconfigurations as the input stream is acted upon by the firsttransformation node 502 and the remainder of the transformations withinthe pipeline are then performed sequentially 502, 503, 504, 505 for theentire pipeline with no introduction of new data internal to the initialnode or splitting output stream prior to last node of the pipeline 505.This configuration is the current state of the art for transformationpipelines and is the most general form of these constructs. Lineartransformation pipelines require no special manipulation to simplify thedata pathway and are thus referred to as non-decomposable they aretherefore processed by the general transformer service 160. The exampledepicted in this diagram was chosen to convey the configuration of alinear transformation pipeline and is the simplest form of theconfiguration felt to show the point. It in no way implies limitation ofthe invention.

FIG. 6 is a process flow diagram of a method 600 for one embodiment of atransformation pipeline where one transformation node 607 in atransformation pipeline receives data streams from two sourcetransformation nodes 601. The invention handles this transformationpipeline configuration by decomposing or serializing the input events602-603, 604-605 heavily relying on post transformation functioncontinuation. The results of individual transformation nodes 602, 604just antecedent to the destination transformation node 606 and placedinto a single specialized data storage transformation node 603, 605(shown twice as process occurs twice). To process this transformationpipeline the pipeline must be taken apart and is handled by thedecomposable transformer service 150. The combined results thenretrieved from the data store 606 and serve as the input stream for thetransformation node within the transformation pipeline backbone 607,608. The example depicted in this diagram was chosen to convey theconfiguration of transformation pipelines with individual transformationnodes that receive input from two source nodes 602, 604 and is thesimplest form of the configuration felt to show the point. It in no wayimplies limitation of the invention. One knowledgeable in the art willrealize the great number of permutations and topologies possible,especially as the invention places no design restrictions on the numberof transformation nodes receiving input from greater than one sources orthe number sources providing input to a destination node.

FIG. 7 is a process flow diagram of a method 700 for one embodiment of atransformation pipeline where one transformation node 703 in atransformation pipeline sends output data stream to two destinationtransformation nodes 701, 706, 708 in potentially two separatetransformation pipelines. The invention handles this transformationpipeline configuration by decomposing or serializing the output events704,705-706, 707-708. The results of the source transformation node 703just antecedent to the destination transformation nodes 706 and placedinto a single specialized data storage transformation node 704, 705, 707(shown three times as storage occurs and retrieval occurs twice). Theresults of the antecedent transformation node may then be retrieved froma data store 704 and serves as the input stream for the transformationnodes two downstream transformation pipeline 706, 708. The exampledepicted in this diagram was chosen to convey the configuration oftransformation pipelines with individual transformation nodes that sendoutput streams to two destination nodes 706, 708 and is the simplestform of the configuration felt to show the point. It in no way implieslimitation of the invention. One knowledgeable in the art will realizethe great number of permutations and topologies possible, especially asthe invention places no design restrictions on the number oftransformation nodes sending output to greater than one destination orthe number destinations receiving input from a source node. This exampletransformation pipeline is also complex and must be disassembled toprocess fully, it would therefore also be processed by the decomposabletransformation server 150.

FIG. 8 is a block diagram 800 of websites on the world wide web that areexample target types of a distributed system for large volume extractionof deep web data. www.seismi.org 810 is a website of geoseismic datawhich by nature is non-textual and therefore has very few tags thatmight be useful to conventional web crawlers. Data retrieved from thistype of web site also does not fit well into a relational data storesetting and might require extensive post-scrape transformation beforestorage in a document type data store.theunitedstates.io/federal_spending/ 811 is a web site that publishesraw spending data reports with are largely textual, but has extremelyfew, if any, web related tags and is thus poorly indexed or retrieved byconventional scraping. This type of web site also is expected to have avery large volume of data which again serves to thwart conventional webcrawling tools. Further, the raw spending data might require significantpre-processing prior to meaningful data store storage.toolkit.climate.gov 812, like www.seismi.org 810, is a site that wouldbe expected to have large amounts of non-textual climate data that needsto be processed with few if any web related tags meaning that climateintrinsic keywords would need to be employed for meaningful retrieval ofthe scraped data and, again both data transformation steps andpre-storage processing may be needed prior to meaningful storage.http://hall-of-justice.herokuapp.com/category/corrections/ 813,http://hall-of-justice.herokuapp.com/category/financial/ 814, andhttp://www.electionpassport.com 815 are all similar in that they aresites with extremely large volumes of free form textual data with few ifany web tags and high probability that data retrieved will need to beprocessed prior to output or storage.

FIG. 9 is a process flow diagram of a method 900 for a high volume webcrawling module 115. Parameters for one or more scrape campaigns,configuration data which may be comprised of, but is not limited to: websites or web pages to be traversed, keywords or tags for web documentdata to be parsed, and search expansion rules for following links orother references found on the sites scraped, as well as any other spiderconfiguration information included by the authors of the scrapecampaign; and scrape campaign control directives which may include butwould not be limited to: the number of spiders to be used in thecampaign, relative resource usage priorities for specific web sites orpages within the intended scrape campaigns, directives for adjustmentsto be made to the scrape campaign upon the encounter of specific resultsor types of results, directives for application of specific scrapecampaign result data pre-processing and post-processing steps and outputformat directives including persistent storage formalization rules; arereceived through either a command line interface 910 which may receivecommands either from an interactive terminal 105 or another softwareapplication on a computing system 115 or from software applications 110through a HTTP based RESTful JSON application programming interface(API) 920. The use of REST and JSON within the API should not beconstrued to mean that the invention is dependent on use of only thoseprotocols for this task as one knowledgeable in the art will realizethat any similar protocols such as, but not limited to, MQTT-basedmessaging, SOAP or AJAX could be employed. The use of REST and JSON isonly in accordance with current practice and inventor decision. Scrapecampaign control and spider configuration parameters received areformalized, as necessary and stored in data store for future use whenthe scrape campaign is initiated. In initiation may be immediate ordelayed and the same scrape campaign may be repeatedly run as parameterspersist until purged. One knowledgeable in the art will comprehend thatkey-value data stores such as Redis are very well suited for storage ofscrape campaign parameter data, however, the invention does not dictatethe use of any specific type of data store for scrape campaign data.Once the command to initiate the scrape campaign is received, theinvention uses the control directives passed to it by the scrapecampaign authors to coordinate the scrape campaign 940. Directives froma list comprising the number and complexity of the web sites to bescraped, the priorities assigned to specific web sites or pages, thenumber different spider configurations to be employed, the speed theauthor desires the scrape to progress among other factors are used todetermine the number of spiders that will be deployed and the number ofscrape servers to be included in the scrape as per predeterminedprogramming within the invention. While the scrape is active, progressand operational information such as stuck spiders and intermediatescrape results is continuously monitored 950 by the scrape campaigncontroller module through the scrape controllers 115 such that theauthors of the scrape campaign can determine the progress made in thescrape, have some indication of what results have been produced, knowwhat tasks the spiders still have pending as well as any links that mayhave been followed and the impact on the scrape as a whole of thoseadditions as per pre-programmed reporting parameters 980. Monitoring 950and reporting 980 aware of operational issues that have arisen, if any.Monitoring data is logged to a data store 930 for future analysis.Program design of the invention allows for adjustments to the scrapecampaign be made, either due to the just disclosed progress andoperational health reports, or other unforeseen factors, without havingto shut down the running scrape and without loss of previously accruedscrape results 960. Raw scrape results obtained by the individualspiders are passed through the scrape controller modules 115 of thescrape servers 115 and are aggregated and then possibly transformed inspecific ways depending on the predetermined goals of the scrapecampaign 970. The invention offers pre-programmed algorithm toolsets forthis purpose and also offers API hooks that allow the data to be passedto external processing algorithms prior to final output in a formatpre-decided to be most appropriate for the needs of the scrape campaignauthors. Result data may also be appropriately processed and formalizedfor persistent storage in a document based data store 990 such asMongoDB, although, depending on the needs of the authors and the type ofdata retrieved during the scrape, any NOSQL type data storage or even arelational database may be used. The invention has no dependency for anyparticular data store type for persistent storage of scrape results.

Special mention should be made concerning the spiders used in theinvention. The authors chose to use Scrapy (Scrapinghub, LTD.,www.scrapy.org), a free, open source, BSD licensed, web crawlingframework, to generate the spiders employed in web scrapes coordinatedby the invention. Scrapy was chosen for several reasons some of whichare: The programming, in Python, for the function of a basic web spideris already present and scrape authors therefore do not need extensiveprogramming expertise in designing spiders to use the framework; Theformat and keywords for the remaining configuration parameters 900needed to create scrape campaign specific spiders is well defined,feature robust and well documented(http://doc.scrapy.org/en/latest/index.html), and the Scrapy frameworkhas been shown reliable and stable during use by such high datathroughput web sites as CareerBuilder.com, BiteFinder.com andData.gov.uk. While the invention currently makes integral use of theScrapy framework for the definition of spiders used, it is notprogrammatically dependent on the Scrapy framework to the point thatanother web crawling agent framework (e.g. OXPath—http://oxpath.org)could not be substituted if a better alternative were to be found andthe use of Scrapy should not be seen to strictly define the invention inthat capability.

FIG. 10 is a listing of a very simple example Scrapy web spiderconfiguration file 1000. This listing requires that the Scrapy frameworkas well as libraries on which Scrapy depends(http://doc.scrapy.org/en/1.0/intro/install.html) are present on thesystem running the web scrape. While highly simplified, the listing 1000shows all of the major sections needed to create a scrape specificspider 1010, 1020, 1030, 1040 At the top of the listing 1010 is found asection that declares the portions of the Scrapy framework that is to beincluded in the creation of the current spider. Going down the listing,the next section 1020 declares a name to be used to identify this spidertype as well as the world wide web domains the spider is allowed totraverse during the scrape and last, the url of the starting point ofthe scrape. In the next section 1030 are any rules to apply whenencountering HTML links during a scrape and also what algorithms shouldbe used when processing the target information of the scrape, in thiscase the spider is scraping specific types of HTML links from theexample.com domain. The last section 1040 has the instructions on how toprocess the target data, including instructions for data associated tospecific web tags. While the spider created by this sample configurationwould have limited capability, it is functional and would, as writtencomplete its scrape. One will immediately appreciate that all of thedirectives in the listed spider definition have to do with retrievingthe data and not the minutia of how the spider gets to the web site orimplements the instructions given in the listed file, etc. This providesthe rationale for this framework being used in the invention.

FIG. 11 is a method flow diagram showing an exemplary method 1100 usedin the capture and storage of time series data from sensors withheterogeneous reporting profiles according to an embodiment of theinvention. In the first step of the method 1105, data is received from aset of sensors connected to a capture and analysis device as in theembodiment. The sensor data received might be captured and stored undertwo main paradigms. One is that the sensor data arrives at a defined,reliable periodicity, which may be continuously, but the amount of dataper unit time is reliably homogeneous and thus the capture and storageof the sensor data is easy to perform using simple time based models.This paradigm and its resolution is prior art and is not depicted. Thesecond paradigm occurs when the sensors being monitored send data atirregular intervals and the amount of data received by the capture andanalysis device can vary greatly overtime. This heterogeneous sensordata behavior demands different processing strategies than does thehomogeneous counterpart. Sensor data capture devices that store sensordata at strictly regular time intervals fair badly as the amounts ofdata per storage cycle can vary greatly. Two strategies that have beenfound to work reliably in conditions of heterogeneous data influx areevent driven and stream capture. The event driven strategy holds data inthe memory of a data stream management engine 120 until a preset numberof data events have occurred 1110-1120. Data is processed by selectingthe parameters, or dimensions within it that are of importance to theadministrator and then stored to the data store when a predeterminedthreshold of events is reached 1120, 1130. The streaming strategy usesthe quantity of data accumulated in a data stream management engine 120as the trigger 1110-1115 to commit the processed sensor data to storage1115-1130. According to the embodiment, an administrator may preselecteither event driven or stream driven commitment, as well as many otherparameters pertaining to analysis of sensor data using theadministration device 120.

Under conditions of heterogeneous sensor data transmission, there willbe times when the rate at which the incoming data to be committed todata store, exceeds the transmission capacity of a single data swimlane1125. This possibility is accounted for by allowing the system totransparently assign more than one real swimlane to a single datatransfer. For example if a single real swimlane can transfer 5 sensorsworth of data per unit time and the data from 8 sensors must becommitted in that unit time, the system can, if pre-set by theadministrator, a metaswimlane, illustrated in FIG. 12 can use real 2swimlanes, one to transfer 5 sensors worth of data to the data store andthe other to transfer three sensors' worth of data to the data store,maintaining the appearance that a single swimlane is in use to thecommitting process.

All sensor data, regardless of delivery circumstances are stored in amultidimensional time series data store 1130 which is designed for verylow overhead, rapid data storage and minimal maintenance needs to sapresources. The embodiment uses a key-value pair data store examples ofwhich are RIAK™, REDIS™, and BERKELEY DB™ for their low overhead andspeed although the invention is not specifically tied to a single datastore type that is known in the art should another with better responseto feature characteristics emerge. Due to factors easily surmised bythose knowledgeable in the art, data store commitment reliability isdependent on data store data size under the conditions intrinsic to timeseries sensor data analysis. The number of data records must be keptrelatively low for the herein disclosed purpose. As an example one groupof developers restrict the size of their multidimensional time serieskey-value pair data store to approximately 8.64×10⁴ records, equivalentto 24 hours of one-second interval sensor readings or 60 days ofone-minute interval readings. In this development system the oldest datais deleted from the data store and lost. This loss of data is acceptableunder development conditions but in a production environment, the lossof the older data is almost always significant and unacceptable. Theinvention accounts for this need to retain older data by stipulatingthat aged data be placed in long term storage. In the embodiment, thearchival storage is included 1170. This archival storage as shownprovided by data archive 120 might be locally provided by the user,might be cloud based such as that offered by Amazon Web Services orGoogle or could be any other available very large capacity storagemethod known to those skilled in the art. Sensor data can bespecifically retrieved, using complex query logic 1135 and transformedusing such tools as mean reading of all query included sensors, varianceof all readings of all sensors queried, standard deviation of queriedsensors and more complex types such as standard linear interpolation,Kalman filtering and smoothing, may be applied. Data can then berepresented in various formats such as, but not limited to text, JSON,KML, GEOJSON and TOPOJSON by the system depending on the ultimate use ofthe resultant information 1180.

FIG. 12 is a process flow diagram of a method for the use ofmetaswimlanes to transparently accommodate levels of data streamingwhich would overload a single swimlane according to an embodiment of theinvention. As previously described, when attempting to commit data fromsets of real time sensors that send data at irregular time intervals andprobably heterogeneous amounts per unit time, it is likely thatsituations will arise when the instantaneous influx of data to betransferred from a data stream management engine 1210, also shown incontext to an entire embodiment of the invention as 120 to amultidimensional time series data store 1220 shown in context as 120 ina system embodiment of the invention 100, will exceed the instantaneousdata capacity of a single data channel, or swimlane 1211 a between thedata stream management engine 1210 and the multidimensional time seriesdata store 1220. Under those conditions, if a remedy could not bebrought to bare, important, possibly crucial data could be lost. Theremedy taken and shown in this embodiment is the ability of the systemto, when configured, combine the transfer and commitment bandwidth oftwo or more real swimlanes 1215 a, 1215 b in a way that is transparentto the committing process. This means that the invention handles thephysical transfer pathway as well as the logical details such astracking the multiple key-value pairs, process identifications and anyapplication specific bookkeeping involved as overhead to the process andthen creating a data structure to have the data records act as a singleentity in subsequent data manipulations.

FIG. 13 is a simplified example of the use a Kalman filter to extractand smooth estimated system state from noisy sensor data according to anembodiment of the invention. Because of its ability to extract reliablyaccurate, interpretable data in cases of noisy input data, heavy use ismade of Kalman filters in data transformation functions of variousembodiments of the invention. It is useful to provide a simpledemonstration of how such filters might work in one or more embodimentsof the invention. For exemplary purposes, let us imagine that miners ina deep underground mine dig into a large underground repository ofcarbon dioxide, which rushes into the lowest level of the minedisplacing a significant amount of the atmosphere in the mine's lowestlevel; assume the CO₂ level there stabilizes at 50%. The mine companydecides to use a combination of lithium hydroxide canisters and the mineshaft's ventilation to handle the problem. A system 120 according to theinvention, may be connected to an array of CO₂ sensors to monitor theprogress of the cleanup. In the example, 50% CO₂ registers as 1000 onthe CO₂ sensors and as a whole the manufacturer states the array willhave a noise level of 400. It is believed that the efforts can remove15.0% of the present CO₂ per hour.

Looking at the Kalman filter equations listed in 1310:

a is equal to the percent of CO2 that will be left, compared to thepercent in the previous measurement period or 100%-15%=85%. So a=0.85.

^x_(k) represents an estimated current result

^x_(k-1) represents the previous estimated result

r is the publish noise level of the sensor or sensor array

z_(k) represents the current observed result

p_(k) is the prediction error between the last previous expected resultand the last previous observed result.

Lastly, gk is the factor by which the difference between the lastexpected result and the current observed result that when added to thelast expected result will produce the current expected result.

For each data point plotted in analysis of the sensor data the expectedresults are calculated using the top equation in 1310 and the errorprediction using the second equation. The lower equations are used toupdate the numbers used to calculate the next set of estimated values^x_(k). Looking at 1320, 1320 e shows calculated CO₂ values (x_(k))determined by multiplying the previous CO₂ expected value by theexpected reduction of 15% (a) 1330 a. 1320 d are the actual valuesreported by the CO₂ sensors 1330 b at the displayed time points 1320 f.The Kalman estimated values, starting at the initial reported CO₂ sensorvalue is shown 1320 b. When graphed, this set of estimated values isdepicted in line 1330 c. The effect of Kalman filter smoothing can beseen by comparing the graphed actual CO₂ sensor readings 1330 b to thegraphed Kalman filter data 1330 c. As the embodiment stores data longterm, users of it can also take advantage of a variant of the Kalmanfilter known as Kalman soothing where data from an another time periodis used to better visualize current data. An example would be to usecorrection data from hours four through eight of the CO₂ analysis tosmooth hours 12 thorough 14, not depicted.

FIG. 18 is a method process flow diagram showing the operation of anautomated planning service module according to an embodiment of theinvention. The analytics data results from the system are supplied tothe automated planning service module 1802 as depicted in 130. Withinthe module the analytic data results are mapped to all possible businessactions or decisions which are suggested by the broad findings and knownwithin the system. Many of these actions may have been enteredspecifically for the current campaign 1802. Any external sourceinformation such as existing business practices that impact thedecision, legal and regulatory considerations that impact the proposedaction among an additional plurality of possible factors known to theart, are then incorporated into the action selection process 1803. Oncethe broadest set of possible prospective actions accounting for externalparameters is known, information theory statistics algorithms andmachine learning principles are employed on the analytic data developedby the system 100 to reliably predict the probable outcomes of pursuingeach choice and provide statistical data associated with each action1804. The data pertaining to actions with a favorable outcome valueabove a predetermined threshold are sent to the simulation module 125and the observation and state estimation 140 modules for appropriatepresentation to end users as dictated by the authors of the relatedanalytical campaign 1805.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be describedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, or other appropriatecomputing device), a consumer electronic device, a music player, or anyother suitable electronic device, router, switch, or other suitabledevice, or any combination thereof. In at least some embodiments, atleast some of the features or functionalities of the various embodimentsdisclosed herein may be implemented in one or more virtualized computingenvironments (e.g., network computing clouds, virtual machines hosted onone or more physical computing machines, or other appropriate virtualenvironments).

Referring now to FIG. 14, there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one embodiment, a computing device 10 may beconfigured or designed to function as a server system utilizing CPU 12,local memory 11 and/or remote memory 16, and interface(s) 15. In atleast one embodiment, CPU 12 may be caused to perform one or more of thedifferent types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 13 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a specific embodiment,a local memory 11 (such as non-volatile random access memory (RAM)and/or read-only memory (ROM), including for example one or more levelsof cached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QualcommSNAPDRAGON™ or Samsung EXYNOS™ CPU as are becoming increasingly commonin the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 15 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity A/V hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown and described above illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 13 handles communicationsas well as routing computations, while in other embodiments a separatededicated communications processor may be provided. In variousembodiments, different types of features or functionalities may beimplemented in a system according to the invention that includes aclient device (such as a tablet device or smartphone running clientsoftware) and server systems (such as a server system described in moredetail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 16 and local memory 11) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 16 or memories 11,16 may also be configured to store data structures, configuration data,encryption data, historical system operations information, or any otherspecific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 15,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 20 includes processors 21that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 24. Processors 21 may carry out computinginstructions under control of an operating system 22 such as, forexample, a version of Microsoft's WINDOWS™ operating system, Apple's MacOS/X or iOS operating systems, some variety of the Linux operatingsystem, Google's ANDROID™ operating system, or the like. In many cases,one or more shared services 23 may be operable in system 20, and may beuseful for providing common services to client applications 24. Services23 may for example be WINDOWS™ services, user-space common services in aLinux environment, or any other type of common service architecture usedwith operating system 21. Input devices 28 may be of any type suitablefor receiving user input, including for example a keyboard, touchscreen,microphone (for example, for voice input), mouse, touchpad, trackball,or any combination thereof. Output devices 27 may be of any typesuitable for providing output to one or more users, whether remote orlocal to system 20, and may include for example one or more screens forvisual output, speakers, printers, or any combination thereof. Memory 25may be random-access memory having any structure and architecture knownin the art, for use by processors 21, for example to run software.Storage devices 26 may be any magnetic, optical, mechanical, memristor,or electrical storage device for storage of data in digital form (suchas those described above). Examples of storage devices 26 include flashmemory, magnetic hard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 16, there is shown a blockdiagram depicting an exemplary architecture 30 for implementing at leasta portion of a system according to an embodiment of the invention on adistributed computing network. According to the embodiment, any numberof clients 33 may be provided. Each client 33 may run software forimplementing client-side portions of the present invention; clients maycomprise a system 20 such as that illustrated above. In addition, anynumber of servers 32 may be provided for handling requests received fromone or more clients 33. Clients 33 and servers 32 may communicate withone another via one or more electronic networks 31, which may be invarious embodiments any of the Internet, a wide area network, a mobiletelephony network (such as CDMA or GSM cellular networks), a wirelessnetwork (such as WiFi, Wimax, LTE, and so forth), or a local areanetwork (or indeed any network topology known in the art; the inventiondoes not prefer any one network topology over any other). Networks 31may be implemented using any known network protocols, including forexample wired and/or wireless protocols.

In addition, in some embodiments, servers 32 may call external services37 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 37 may take place, for example, via one or morenetworks 31. In various embodiments, external services 37 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 24 are implemented on a smartphone or other electronicdevice, client applications 24 may obtain information stored in a serversystem 32 in the cloud or on an external service 37 deployed on one ormore of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 33 or servers 32 (or both)may make use of one or more specialized services or appliances that maybe deployed locally or remotely across one or more networks 31. Forexample, one or more databases 34 may be used or referred to by one ormore embodiments of the invention. It should be understood by one havingordinary skill in the art that databases 34 may be arranged in a widevariety of architectures and using a wide variety of data access andmanipulation means. For example, in various embodiments one or moredatabases 34 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, Hadoop Cassandra, Google BigTable, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 36 and configuration systems 35. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 36 orconfiguration system 35 or approach is specifically required by thedescription of any specific embodiment.

FIG. 17 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to keyboard 49, pointing device 50,hard disk 52, and real-time clock 51. NIC 53 connects to network 54,which may be the Internet or a local network, which local network may ormay not have connections to the Internet. Also shown as part of system40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for automated business decision optimization, comprising: a network-connected computing device comprising a memory and a processor; a high volume deep web scraper comprising first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the network-connected computing device to perform deep web searches by extracting information from data stores located on the Internet that are not accessible by conventional web crawlers; a business data retrieval engine comprising second plurality of programming instructions_stored in the memory of, and operating on the processor of, the network-connected computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the network-connected computing device to: receive a business analysis campaign configuration comprising analysis parameters for optimizing a business decision; retrieve a plurality of business operations data from devices within a business computer network related to the business analysis campaign configuration; direct the high volume deep web scraper to retrieve a plurality of supplemental data from deep web extraction related to the business analysis campaign configuration; a directed computational graph module comprising a third plurality of programming instructions stored in the memory of, and operating on the processor of, the network-connected computing device, wherein the third plurality of programming instructions, when operating on the processor, cause the network-connected computing device to: construct a directed computational graph from the business analysis campaign configuration, wherein: the directed computational graph comprises nodes representing data transformations and edges representing messages between the nodes; and the nodes and edges of the directed computational graph represent data processing pipelines for analyzing the business analysis campaign configuration; and a business decision and business action path simulation engine comprising a fourth plurality of programming instructions stored in the memory of, and operating on the processor of, the network-connected computing device, wherein the fourth plurality of programming instructions, when operating on the processor, cause the network-connected computing device to: determine a set of possible prospective actions from the business analysis campaign configuration, the business operations data, and the supplemental data; simulate the outcome of each prospective action using the data processing pipelines of the directed computing graph as a simulation model; determine an optimal outcome from the parametric analysis by matching the outcome of each prospective action against the analysis parameters; and recommend the prospective action with the optimal outcome as the business decision.
 2. The system of claim 1, wherein the business information retrieval engine employs a portal for human interface device to input the current business analysis campaign configuration.
 3. The system of claim 2, wherein high volume deep web scraper receives at least some scrape control and spider configuration parameters from a customizable cloud-based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further scrape control and spider configuration parameters, controls and monitors the spiders on distributed scrape servers, receives raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign data.
 4. The system of claim 3, wherein the scrape control and spider configuration parameters are provided by a library of spider templates and individual spiders are created using the spider templates.
 5. The system of claim 3, wherein one or more of the business analysis campaign configurations is persistently stored and can be reused or used as basis for similar business analysis campaigns.
 6. The system of claim 1, wherein the business information retrieval engine employs a multidimensional time series data store comprising a fifth plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the fifth plurality of programming instructions, when operating on the processor, cause the network-connected computing device to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data, or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key-value pair with very little or no data transformation from how the aggregated sensor data is received.
 7. The system of claim 1, wherein the directed computational graph module: retrieves streams of input from one or more of a plurality of data sources; filters data to remove data records from each stream based on one or more of the following: absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record; splits filtered data stream into two or more identical streams; formats data within one data stream based upon a set of predetermined parameters; and sends identical data stream for further analysis and either linear transformation or branching transformation using resources of the system.
 8. A method for automated business decision optimization, comprising the steps of: receiving a business analysis campaign configuration comprising analysis parameters for optimizing a business decision; retrieving a plurality of business operations data from devices within a business computer network related to the business analysis campaign configuration; directing a high volume web crawler to retrieve a plurality of supplemental data from deep web extraction related to the business analysis campaign configuration by extracting information from data stores located on the Internet that are not accessible by conventional web crawlers; constructing a directed computational graph from the business analysis campaign configuration, wherein: the directed computational graph comprises nodes representing data transformations and edges representing messages between the nodes; and the nodes and edges of the directed computational graph represent data processing pipelines for analyzing the business analysis campaign configuration; determining a set of possible prospective actions from the business analysis campaign configuration, the business operations data, and the supplemental data; simulating the outcome of each prospective action using the data processing pipelines of the directed computing graph as a simulation model; determining an optimal outcome from the parametric analysis by matching the outcome of each prospective action against the analysis parameters; and recommending the prospective action with the optimal outcome as the business decision.
 9. The method of claim 8, wherein a portal for human interface device is used to input the business analysis campaign configuration.
 10. The method of claim 9, wherein the high volume deep web scraper receives at least some scrape control and spider configuration parameters from a customizable cloud-based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further scrape control and spider configuration parameters, controls and monitors the spiders on distributed scrape servers, and receives raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign data.
 11. The method of claim 10, wherein the scrape control and spider configuration parameters are provided by a library of spider templates and individual spiders are created using the spider templates.
 12. The method of claim, 10 wherein one or more of the business analysis campaign configurations is persistently stored and can be reused or used as basis for similar business analysis campaigns.
 13. The method of claim 9, wherein the a multidimensional time series data is used to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key-value pair with very little or no data transformation from how the aggregated sensor data is received.
 14. The method of claim 8, wherein the directed computational graph module: retrieves streams of input from one or more of a plurality of data sources; filters data to remove data records from the stream based on one or more of the following: absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record; splits filtered data stream into two or more identical streams, formats data within one data stream based upon a set of predetermined parameters so as to prepare for meaningful storage in a data store; and sends identical data stream for further analysis and either linear transformation or branching transformation using resources of the system. 